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DETAILED ACTION 

Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1 .17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1 .1 7(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 
2/27/2007 has been entered. 

Response to Amendment 

2. The amendment filed on 1/30/2007 has been entered. 

The amendment filed 1/30/2007 is objected to under 35 U.S.C. 132(a) because it 
introduces new matter into the disclosure. 35 U.S.C. 132(a) states that no amendment 
shall introduce new matter into the disclosure of the invention. The added material 
which is not supported by the original disclosure is as follows: the changes made to 
paragraph [0006] enters new matter because the "object" is not clearly limited to the 
stylus discussed at paragraph [0028]. 

Applicant is required to cancel the new matter in the reply to this Office Action. 
Information Disclosure Statement 

3. The translation of the Hirokazu Kato, Mark Billinghurst, Koichi Asano, and 
Keihachiro Tachibana article titled An Augmented Reality System and its Calibration 
based on Marker Tracking filed on 3/28/2007 has been considered and the article 
submitted on 1/30/2007 is cited on the attached PTO-892. 
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Response to Arguments 

4. Applicant's arguments filed 1/30/2007 have been fully considered but they are 
not persuasive. In the paragraph spanning pages 1 1 and 12 applicant makes 
reference to applicants specification at paragraph [0005] and argues that applicant 
creates the constraining shape dynamically in accordance with the user's instruction 
when said user operates the virtual image. Applicant in the sentence which spans 
pages 12 and 13 states the Kitamura discloses that the constraining shape as constraint 
condition is calculated base on the geometric model of the object, which is kept in 
advance, and does not even suggest creating the constraining shape in accordance 
with the instruction of a user who manipulates the virtual object, as shown in the present 
invention. However, claim 10 does not claim dynamically creating the constraining 
shape. In view of page 135 first paragraph under the heading of 5 Consolidated 
Manipulation Environment of Kitamura the same 6 DOF tracker is used to control the 
virtual object and to control the position of the constraining real objects which is similar 
to applicants system where stylus 1060 is used to control the virtual objects and to 
control the location of the constraining objects, see applicants paragraphs [0028], 
[0030], and [0041]. Therefore, Kitamura still anticipates the claims and if the claims 
were amended to claim the argued dynamic aspect then Kitamura's statement at page 
1 35 first paragraph under heading 5 "The user employs a six degrees of freedom 
(DOF) tracker device to manipulate virtual objects. The shapes of the real objects are 
assumed to be know in advance, and the motion of the real objects is also obtained by 
the 6 DOF tracker." would still anticipate a claim to dynamically changing a constraining 
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object's location which would dynamically change the entire constraining shape formed 
by the sum of the constraining shapes. Currently amended claim 1 0 claims "obtaining 
a constraining shape based on the obtained three-dimensional position information" and 
in view of the word "obtaining" this claim limitation is met by moving for example the 
cubes of figure 2 to different locations to form a constraining shape of the cubes 
different than before the cubes were moved. Applicant needs to be more specific about 
how the constraining shape is obtained. Currently amended independent claims 1 , 5, 
and 16 do not overcome the Kitamura article for similar reasons. 

Claim Rejections - 35 USC §112 

5. The following is a quotation of the first paragraph of 35 U.S.C. 1 1 2: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

6. Claims 1-3, 5-9, and 16-18 are rejected under 35 U.S.C. 112, first paragraph, as 
failing to comply with the written description requirement. The claim(s) contains subject 
matter which was not described in the specification in such a way as to reasonably 
convey to one skilled in the relevant art that the inventor(s), at the time the application 
was filed, had possession of the claimed invention. The amendment to independent 
claims 1 , 5, and 16 "input three-dimensional position information of a plurality of 
positions inputted by moving a real object in the real space by a user" is not conveyed 
by the originally field disclosure because an object is different than the stylus discussed 
at paragraph [0028] that would be moved by the user. Currently applicant is claiming to 
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move any real object while the application only supports moving stylus 1060. Figures 2, 
10, and 12-14 shows stylus 1060 but do not convey that any real object may be moved 
to input three-dimensional position information. 



Claim Rejections - 35 USC § 102 

7. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

8. Claims 1-3 and 5-18 are rejected under 35 U.S.C. 102(b) as being anticipated by 
Yoshifumi Kitamura and Fumio Kishino, Consolidated Manipulation of Virtual and Real 
Objects, September 1997, Proceedings of the ACM symposium on Virtual reality 
software and technology, pages 133-138. Kitamura teaches an augmented reality 
system that uses object constraints to control the visual interaction between the virtual 
objects and the real objects. 

A detailed analysis of the claims follows. 
Claim 10: 

Kitamura teaches an information processing method for changing the position 
and orientation of a virtual object in mixed reality space obtained by combining a real 
space and a virtual space, said method comprising the steps of: 
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obtaining three dimensional position information of a plurality of positions 
designated by an operating unit moved by a user in the real space (In view of page 135 
first paragraph under the heading of 5 Consolidated Manipulation Environment of 
Kitamura the same 6 DOF tracker is used to control the virtual object and to control the 
position of the constraining real objects which is similar to applicants system where 
stylus 1060 is used to control the virtual objects and to control the location of the 
constraining objects, see applicants paragraphs [0028], [0030], and [0041]. The 6 DOF 
has a stylus the user moves which meets the operating unit moved by a user in the real 
space claim limitation.); 

obtaining a constraining shape based on the obtained three-dimensional position 
information (This claim limitation is met by moving for example the cubes of figure 2 to 
different locations to form a constraining shape of the cubes different than before the 
cubes were moved. Applicant needs to be more specific about how the constraining 
shape is obtained.); 

changing the position and orientation of the virtual object according to 
instructions from the user, based on the obtained constraining shape as constraint 
condition (The first paragraph in section 5 on page 135 describes the user using a 6 
DOF tracker device to manipulate the virtual objects. The shape of the real objects are 
used to constrain the movement of the virtual object by giving the real object a shape 
that the virtual object interacts with in a constrained manner. Sections 5. 1 to 5.4 
discusses manipulation of the virtual object based upon constraint conditions based on 
the shape of the real object. The shape generated from the external instructions 
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constraints the interaction of the virtual object with the real object, see sections 5. 1 to 
5.4.); and 

combining an image of the virtual object generated according to the changed 
position and orientation, and the real image (The introduction on page 133 second full 
paragraph discusses augmented reality which synthesizes a virtual object with a real object. 
The claim does not limit the form of the real image, thus, Kitamura teaches this limitation. 
Section 5. 1 discusses after movement is detected by the 6DOF manipulator the virtual object 
is moved according to the constraints.). 

Claim 11: 

Kitamura teaches an information processing method according to Claim 10, 
further comprising the step of combining a virtual image indicating the constraining 
shape with the real image (The introduction on page 133 second full paragraph discusses 
augmented reality which synthesizes a virtual object with a real object. The claim does not 
limit the form of the real image, thus, Kitamura teaches this limitation. Section 5. 1 discusses 
after movement is detected by the 6DOF manipulator the virtual object is moved according to 
the constraints.). 

Claim 12: 

Kitamura teaches an information processing method according to Claim 10, 
wherein the constraining shape is a plane. (On page 136 in the text above figure 2 determining a 
plane and using the plane to constrain movement of the virtual object is discussed with regards to figure 
2.). 
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Claim 13: 

Kitamura teaches an information processing method according to Claim 10, 
wherein said changing step changing the position and orientation of the virtual object is 
carried out by changing the position and orientation of the operating unit (The 6 DOF 
tracker is an operating unit. The user using the 6 DOF tracker device manipulates the 
virtual objects.). 

Claim 14: 

Kitamura teaches a computer program product comprising a computer readable 
medium storing computer program code for performing the information processing 
method according to Claim 10 wherein the information processing method is executed 
by a computer device (This article is directed to computers that generate the augmented 
reality scene since it was published by ACM for a symposium on virtual reality software 
and technology and since at page 133 in the last sentence in the second paragraph of 
section 1 software/hardware is discussed. Software causing a computer to perform 
Kitamura's augmented reality is inherently stored in a computer readable recording 
medium.). 

Claim 15: 

Kitamura teaches a computer-readable recording medium, storing the computer 
program according to Claim 14 (Software causing a computer to perform Kitamura's 
augmented reality is inherently stored in a computer readable recording medium.). 
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Claim 16: 

The functions of Kitamura corresponds to the claimed units because the software 
causes the computer to become a unit that performs a process. As seen below 
Kitamura performs the claimed processes. 

Kitamura teaches an information processing device for aiding control operations 
relating to controlling the position and orientation of a virtual object, said device 
comprising: 

an image capturing unit configured to capture a real image in real space (The 
measurements by the user in three dimensional real space, the device for obtaining the 
3-D shape by using a range image, and the device for obtaining the 3-D shape by using 
multiple cameras captures a real image in real space. The three paragraphs found in 
section 2 on page 134 of Kitamura teaches the user measuring the real objects, a device 
measuring the real objects with range in response to the user, or a device measuring the 
real objects with multiple cameras range in response to the user. The claim does not limit 
the form of the real image, thus, Kitamura teaches this limitation.); 

a virtual image generation unit configured to generate a virtual image of a virtual 
object according to the position and orientation of said image capturing unit (Kitamura 
discusses in section 2 in the first paragraph using conventional modeling software after 
precisely measuring the size or length of the real object by hand or by the devices which 
is an generation unit capable of generating three-dimensional positional information.) ; 

a superimposing unit configured to superimpose the generated virtual image with 
the captured real image (Section 2 discusses mixing the virtual and real world objects.); 
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an inputting unit configured to input three-dimensional position information of a 
plurality of positions inputted by a moving a real object in the real space by a user (In 
view of page 135 first paragraph under the heading of 5 Consolidated Manipulation 
Environment ofKitamura the same 6 DOF tracker is used to control the virtual object 
and to control the position of the constraining real objects which is similar to applicants 
system where stylus 1060 is used to control the virtual objects and to control the 
location of the constraining objects, see applicants paragraphs [0028], [0030], and 
[0041]. The 6 DOF has a stylus the user moves which meets the moving a real object 
limitation in the real space by a user claim limitation. Similarly using the 6 DOF to 
sense the new location of a moved real constraining object by the user meets the 
moving a real object limitation in the real space by a user claim limitation.); 

a setting unit configured to set a constraining shape based on the inputting three- 
dimensional position information (The first paragraph in section 5 on page 135 describes 
the user using a 6 DOF tracker device to manipulate the virtual objects. The shape of the 
real objects are used to constrain the movement of the virtual object by giving the real 
object a shape that the virtual object interacts with in a constrained manner.) ; and 

an operating unit configured to control the position and orientation of the virtual 
object based on the constraining shape in accordance with the a user's instruction^ 
Sections 5. 1 to 5.4 discusses manipulation of the virtual object based upon constraint 
conditions based on the shape of the real object. The shape generated from the external 
instructions constrains the interaction of the virtual object with the real object, see 
sections 5. 1 to 5.4. The introduction on page 133 second full paragraph discusses 
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augmented reality which synthesizes a virtual object with a real object. Section 5. 1 discusses 
after movement is detected by the 6DOF manipulator the virtual object is moved according to 
the constraints.). 
Claim 17: 

Kitamura teaches an information processing device according to Claim 16, 
wherein the constraining shape is defined by polygons (Figure 2's Cubes are formed 
with polygons.) and the apexes of the polygons are at positions inputted by the user 
(According to section 5 the when the real objects are moved the 6DOF senses the new 
locations which requires for a cube the apex's position to be inputted by the user.) or the 
constraining shape is a plane passing through the positions inputted by the user (A real 
sensed surface is a planar real world object whose position is inputted by the user. The 
claim does not claim how or when the plane's position is inputted by the user.). 

Claim 18: 

Kitamura teaches an information processing device according to Claim 16, 
wherein said operating unit performs at least one of the following operations in 
performing an operation controlling the position and orientation of the virtual object: 

a translation operation for causing translational movement of the virtual object 
based on the constraining shape (Page 137 column 1 lines 1-3 teaches translation 
when the virtual object collides with the real surface.); or 

a rotation operation for rotating the virtual object on an axis which is a normal 
vector at a plane where the constraining shape and the virtual object come into contact 
(Page 137 column 1 lines 1-3 teaches rotation when the virtual object collides with the 
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real surface.) (Page 137 column 1 lines 1-3 also teaches translation and rotation when 
the virtual object collides with the real surface). 
Claims 1-3: 

Means plus function claims 1-3 correspond to device claims 16-18 and the 
means of Kitamura, software and computer, are equivalent to applicant's means of 
software and computer. 

Claims 5-7: 

Step for claims 5-7 correspond to device claims 16-18 and the steps of Kitamura, 
software and computer, are equivalent to applicant's steps performed by software and 
computer. 

Claims 8 and 9: 

Claims 8 and 9 mirror claims 14 and 15 addressed above and they are rejected 
for the same reasons given above for claims 14 and 15. 

9. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffery A Brier whose telephone number is (571 ) 272- 
7656. The examiner can normally be reached on M-F from 7:30 to 4:00. If attempts to 
reach the examiner by telephone are unsuccessful, the examiner's supervisor, Michael 
Razavi, can be reached at (571 ) 272-7664. The fax phone Number for the organization 
where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
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published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 




